In order to solve the problem of poor robustness of Kernelized Correlation Filter (KCF) in complex scenes, a new object tracking algorithm based on Self-Adaptive Compound Kernel (SACK) was proposed. The tracking task was decomposed into two independent subtasks:position tracking and scale tracking. Firstly, the risk objective function of SACK weight was constructed by using the self-adaptive compound of linear kernel and Gaussian kernel as the kernel tracking filter. The weights of linear kernel and Gaussian kernel were adjusted adaptively by the constructed function according to the response values of kernels, which not only considered the minimum empirical risk function of different kernel response outputs, but also considered the risk function of maximum response value, and had the advantages of local kernel and global kernel. Then, the exact position of object was obtained according to the output response of the SACK filter, and the adaptive update rate based on the maximum response value of object was designed to adaptively update the position tracking filter. Finally, the scale tracker was used to estimate the object scale. The experimental results show that, the success rate and distance precision of the proposed algorithm are optimal on OTB-50 database, which is 6.8 percentage points and 4.1 percentage points higher than those of KCF algorithm respectively, 2 percentage points and 3.2 percentage points higher than those of Bidirectional Scale Estimation Tracker (BSET) algorithm respectively. The proposed algorithm has strong adaptability to complex scenes such as deformation and occlusion.